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PURPOSE: To synthesize an enlarged image without 
lowering resolution. 

CONSTITUTION: Conversion coefficient data to be 
complemented by a conversion coefficient for which 
orthogonal transformation is performed to the image 
data of a reference image divided into blocks and a 
conversion coefficient which is provided by performing 
orthogonal transformation to an enlarged image enlarging 
this reference image, are previously learnt and stored in 
a ROM table 5. The orthogonal transformation is 
performed to the image data of the reference image, 
which is divided into blocks by a blocking part 2, by an 
orthogonal transforming part 3, the conversion 
coefficient caused by this orthogonal transformation is 
classified into classes, and a class code is generated by 
a class code generating part 4. The conversion 
coefficient data corresponding to this class code are 
read from the ROM table 5, sent to a coefficient 
synthesizing part 6, synthesized with the conversion 
coefficient of the reference image and sent to an inverse transformation part 7. At the inverse 
transformation part 7, inverse transformation is performed to the conversion coefficient and the 
image data, which are divided into blocks, of the enlarged image are decomposed into blocks by 
a block decomposition part 8. 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1 This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



CLAIMS 

• 

[Claim(s)] 

[Claim 1] A blocking means to block the image data of the criteria image inputted, An orthogonal 
transformation means to perform orthogonal transformation to the image data blocked from the 
above-mentioned blocking means, and to ask for a transform coefficient, A class classification 
means to generate the class code which classifies the transform coefficient from the above- 
mentioned orthogonal transformation means into two or more classes, and shows this class, A 
multiplier data generating means by which the transform coefficient data corresponding to the 
class code from the above-mentioned class classification means are read, A multiplier 
composition means to compound the transform coefficient of an expansion image with the 
transform coefficient from the above-mentioned orthogonal transformation means, and the 
transform coefficient data from the above-mentioned multiplier data generating means, 
Electronic zoom equipment characterized by consisting of an inverse transformation means to 
perform inverse transformation to a transform coefficient from the above-mentioned multiplier 
composition means, and to output the image data for every block of an expansion image. 
[Claim 2] The electronic zoom equipment according to claim 1 characterized by to ask for a 
transform coefficient, to ask for the transform coefficient data optimized by learning between 
the transform coefficient of the above-mentioned criteria image, and the transform coefficient of 
the above-mentioned expansion image by blocking the image data of a criteria image and an 
expansion image for the above-mentioned multiplier data generating means, respectively, and 
performing orthogonal transformation to it, and for the optimized transform coefficient data 
corresponding to the above-mentioned class code and this class code to be beforehand 
memorized by the table. 

[Claim 3] A block chemically-modified [ which blocks the image data of the criteria image 
inputted ] degree, The orthogonal transformation process which performs orthogonal 
transformation to the image data blocked from the block chemically-modified [ above- 
mentioned ] degree, and asks for a transform coefficient, The class classification procedure 
which generates the class code which classifies the transform coefficient from the above- 
mentioned orthogonal transformation process into two or more classes, and shows this class, 
The multiplier data generating process that the transform coefficient data corresponding to the 
class code from the above-mentioned class classification procedure are read, The multiplier 
composition process which compounds the transform coefficient of an expansion image with the 
transform coefficient from the above-mentioned orthogonal transformation process, and the 
transform coefficient data from the above-mentioned multiplier data generating process, The 
electronic zoom approach characterized by consisting of the inverse transformation process 
which performs inverse transformation to a transform coefficient from the above-mentioned 
multiplier composition process, and outputs the image data for every block of an expansion 
image. 

[Claim 4] The electronic zoom approach according to claim 3 characterized by to ask for a 
transform coefficient, to ask for the transform coefficient data optimized by learning between 
the transform coefficient of the above-mentioned criteria image, and the transform coefficient of 
the above-mentioned expansion image by blocking the image data of a criteria image and an 
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expansion image at the above-mentioned multiplier data generating process, respectively, and 
performing orthogonal transformation to it, and for the optimized transform coefficient data 
corresponding to the above-mentioned class code and this class code to be beforehand 
memorized by the table. 



[Translation done.] 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1 This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Industrial Application] This invention relates to the electronic zoom equipment and the 

electronic zoom approach of outputting the expanded image. 

[0002] 

[Description of the Prior Art] The method of expanding an image exists by extending the field by 
which orthogonal transformation is carried out to the approach of interpolating a pixel in an 
expansion image in space, and expanding an image to the approach of carrying out a zoom of the 
digital image and expanding it with a television camera etc., filling up the transform coefficient 
when performing orthogonal transformation to the image data of an expansion image in 
conversion space in the field by which an escape was carried out [ above-mentioned ], and 
performing inverse transformation. 

[0003] Although the pixel is interpolated in the approach of interpolating the pixel in an 
expansion image, using a linear interpolation filter etc. in the above-mentioned space, resolution 
falls and that dotage is conspicuous poses a problem from the image before the expansion image 
obtained by this approach is expanded on the expansion image. 

[0004] The approach of filling up a transform coefficient and on the other hand performing 
inverse transformation does not reduce the resolution of an expansion image rather than the 
approach of interpolating a pixel using the above-mentioned linear interpolation filter. 
[0005] 

[Problem(s) to be Solved by the Invention] By the way, the high frequency component of an 
image appears in the transform coefficient usually obtained by performing orthogonal 
transformation to image data. However, in the approach of filling up the conventional transform 
coefficient and performing inverse transformation, since the transform coefficient supplemented 
in the expansion image is only made into zero, the high frequency component is not contained. 
Therefore, when these zero perform inverse transformation using the transform coefficient set 
up and filled up and compound an expansion image, the image for which resolution was 
insufficient is obtained. 

[0006] Then, this invention offers the electronic zoom equipment and the electronic zoom 
approach of obtaining an expansion image with the transform coefficient in which the description 
of an image was made to reflect in view of the above-mentioned actual condition including the 
high frequency component of an expansion image. 
[0007] 

[Means for Solving the Problem] A blocking means to block the image data of the criteria image 
into which the electronic zoom equipment concerning this invention is inputted, An orthogonal 
transformation means to perform orthogonal transformation to the image data blocked from the 
above-mentioned blocking means, and to ask for a transform coefficient, A class classification 
means to generate the class code which classifies the transform coefficient from the above- 
mentioned orthogonal transformation means into two or more classes, and shows this class, A 
multiplier data generating means by which the transform coefficient data corresponding to the 
class code from the above-mentioned class classification means are read, A multiplier 
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composition means to compound the transform coefficient of an expansion image with the 
transform coefficient from the above-mentioned orthogonal transformation means, and the 
transform coefficient data from the above-mentioned multiplier data generating means, Inverse 
transformation is performed to a transform coefficient from the above-mentioned multiplier 
composition means, and the technical problem mentioned above by consisting of an inverse 
transformation means to output the image data for every block of an expansion image is solved. 
[0008] Moreover, it asks for a transform coefficient, it asks for the transform coefficient data 
optimized by learning between the transform coefficient of the above-mentioned criteria image, 
and the transform coefficient of the above-mentioned expansion image, and it is characterized 
by for the optimized transform coefficient data corresponding to the above-mentioned class 
code and this class code to be beforehand memorized by the table by blocking the image data of 
a criteria image and an expansion image for the above-mentioned multiplier data generating 
means, respectively, and performing orthogonal transformation to it. 

[0009] A block chemically-modified [ which blocks the image data of the criteria image into 
which the electronic zoom approach concerning this invention is inputted ] degree, The 
orthogonal transformation process which performs orthogonal transformation to the image data 
blocked from the block chemically-modified [ above-mentioned ] degree, and asks for a 
transform coefficient, The class classification procedure which generates the class code which 
classifies the transform coefficient from the above-mentioned orthogonal transformation process 
into two or more classes, and shows this class, The multiplier data generating process that the 
transform coefficient data corresponding to the class code from the above-mentioned class 
classification procedure are read, The multiplier composition process which compounds the 
transform coefficient of an expansion image with the transform coefficient from the above- 
mentioned orthogonal transformation process, and the transform coefficient data from the 
above-mentioned multiplier data generating process, Inverse transformation is performed to a 
transform coefficient from the above-mentioned multiplier composition process, and the 
technical problem mentioned above by consisting of the inverse transformation process which 
outputs the image data for every block of an expansion image is solved. 

[0010] Moreover, it asks for a transform coefficient, it asks for the transform coefficient data 
optimized by learning between the transform coefficient of the above-mentioned criteria image, 
and the transform coefficient of the above-mentioned expansion image, and it is characterized 
by for the optimized transform coefficient data corresponding to the above-mentioned class 
code and this class code to be beforehand memorized by the table by blocking the image data of 
a criteria image and an expansion image at the above-mentioned multiplier data generating 
process, respectively, and performing orthogonal transformation to it. 
[0011] 

[Function] In this invention, an expansion image is compounded by compounding the transform 
coefficient data in the extended field which is beforehand learned by the transform coefficient 
and the multiplier data generating means which carried out orthogonal transformation of the 
image data of a criteria image, and it asked for it, and is memorized, and performing inverse 
transformation. 
[0012] 

[Example] Hereafter, the desirable example of this invention is explained, referring to a drawing. 
The rough configuration of the electronic zoom equipment concerning this invention is shown in 
drawing 1 . 

[0013] The digital image data of the criteria image inputted from the input terminal 1 of drawing 1 
are sent to the blocking section 2. The rate of a zoom of the request which expands a criteria 
image is given from the outside to the above-mentioned blocking section 2. Therefore, in the 
above-mentioned blocking section 2, with the above-mentioned rate of a zoom, the field where it 
is expanded in a criteria image can be pinpointed, and blocking of the image data of this 
pinpointed field that should be expanded is performed. The image data by which blocking was 
carried out [ above-mentioned ] is outputted to the orthogonal transformation section 3 for 
every block. In this orthogonal transformation section 3, orthogonal transformation is performed 
to each image data for every above-mentioned block, and a transform coefficient is obtained. As 
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this orthogonal transformation, it is possible to use DCT (Discrete Cosine Transform), a 
Hadamard transform, KL (karroo NENREBU) conversion, etc. 

[0014] For example, in the above-mentioned blocking section 2, when orthogonal transformation 
is performed to each pixel data blocked in four-line four trains (4x4), a transform coefficient 
group as shown by O mark of (a) of drawing 2 is obtained. 

[0015] Here, the image data blocked in four-line four trains in a criteria image (4x4) is doubled 
two, and when compounding the expansion image expanded to the image data by which eight-line 
eight trains (8x8) were blocked, the transform coefficient shown by x mark of (a) of drawing 2 will 
be filled up. Therefore, the location shown by - mark of (b) of drawing 2 is supplemented with a 
transform coefficient, and the image data by which the expansion image which expanded the 
criteria image twice was blocked can be obtained by performing inverse transformation to all the 
transform coefficients shown by O mark and - mark. 

[0016] The transform coefficient obtained in the above-mentioned orthogonal transformation 
section 3 is sent to the class code generating section 4. In this class code generating section 4, 
the class classification of the transform coefficient for every sent block is carried out, and the 
class code of this transform coefficient by which the class classification was carried out is 
generated. The generated class code is sent to the ROM (read-only memory) table 5. 
[0017] The optimized transform coefficient data with which this ROM table 5 is supplemented in 
an expansion image are beforehand obtained by study, and are memorized as a table with the 
class code. Therefore, the transform coefficient data in the expansion image corresponding to 
this class code are read from the above-mentioned ROM table 5 by using the class code from 
the above-mentioned class code generating section 4 as the address. 

[0018] Here, the transform coefficient data beforehand learned by the above-mentioned ROM 
table 5 are explained. 

[0019] First, for example, the criteria image of (a) of drawing 3 and the expansion image which 
shows the zoom core s of this criteria image to (b) of drawing 3 expanded as a core are 
prepared. It can ask for the image data of this expansion image by the rate calculation approach 
of a zoom proposed in a specification, a drawing, etc. of JP,4-318766,A. 

[0020] The procedure when learning a transform coefficient is shown in the flow chart of drawing 
4 , and at step SI, the image data of the criteria image of (a) of drawing 3 is blocked, orthogonal 
transformation is performed to the image data by which blocking was carried out [ above- 
mentioned ] at step S2, and it asks for a transform coefficient. Furthermore, at step S3, the 
class classification of the above-mentioned transform coefficient is carried out, and the class 
code corresponding to this transform coefficient by which the class classification was carried 
out is generated. 

[0021] Moreover, it is parallel to above-mentioned actuation, and in step S4, the image data of 
the expansion image of (b) of drawing 3 is blocked, orthogonal transformation is performed to the 
image data by which blocking was carried out [ above-mentioned ] at step S5, and it asks for a 
transform coefficient. 

[0022] Next, it progresses to step S6 and learns between the transform coefficient by which the 
class classification was carried out at step S3, and the transform coefficient of the expansion 
image corresponding to this transform coefficient. Then, the class classification of the transform 
coefficient is called for and carried out about all the blocked image data in a criteria image at 
step S7, and it distinguishes whether study was performed between the transform coefficients of 
the image data by which the expansion image was blocked. 

[0023] It learns by blocking the image data of the criteria image with which it returns to step S1 
and step S4, and study is not performed further yet if it becomes and expansion image with 
which the above-mentioned study is performed to no blocked image data, performing orthogonal 
transformation to this blocked image data, and asking for a transform coefficient. However, if the 
above-mentioned study is performed to all the blocked image data, a mapping table will be 
generated from the transform coefficient by which the expansion image corresponding to the 
class code corresponding to the class classification by which study was carried out [ above- 
mentioned ] by progressing to step S8, and this class code was optimized. Thus, into the above- 
mentioned ROM table 5, a predetermined class code and the transform coefficient data 
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corresponding to this class code are learned beforehand, and two or more storage is carried out. 
[0024] The simplest classification approach of the class code memorized in the above- 
mentioned ROM table 5 is an approach which does not process compression etc. to the 
transform coefficient obtained by performing orthogonal transformation to the image data for 
every block of the criteria image by which a class classification is carried out, but uses the 
transform coefficient of the image data for every above-mentioned block as a class code as it is. 
However, when a transform coefficient is a class code as it is, the pattern of a number 
equivalent to the above-mentioned class code of transform coefficients must be memorized, 
and, for that, ROM of a huge capacity is needed. Therefore, when implementability is taken into 
consideration, it is possible to use compressive processing for the transform coefficient of the 
criteria image learned. 

[0025] How to reduce for example, the above-mentioned number of class codes there using 
ADRC (ecad dynamic range coding) can be considered. Originally, ADRC is the accommodative 
re-quantizing method developed for [ to turn ] high efficiency coding VTR (video tape recorder), 
and can express the local pattern of signal level efficiently by the short word length. Therefore, 
when ADRC is used, the number of the effective class codes which saved the property of the 
pattern of a transform coefficient can be reduced. 

[0026] Here, the study approach of a prediction coefficient of having used ADRC for the class 
division in the case of using a prediction coefficient as the above-mentioned transform 
coefficient is explained concretely. 

[0027] Drawing 5 shows the rough configuration at the time of learning the prediction coefficient 
memorized in the above-mentioned ROM table 5. From the input terminal 1 1 of this drawing 5 , 
the transform coefficient obtained by performing orthogonal transformation to the 1 -block image 
data of the criteria image which is an image in front of the blocked zoom is inputted, and the 
transform coefficient obtained by performing orthogonal transformation to the 1 -block image 
data of the blocked expansion image corresponding to the 1 -block image data of the above- 
mentioned criteria image is inputted from an input terminal 12. The transform coefficient of 
these two blocks is sent to the study section 13. In this study section 13, it learns using the 
transform coefficient of the inputted criteria image, and the transform coefficient of an 
expansion image, and the prediction coefficient corresponding to a class code and this class 
code is created. This class code and prediction coefficient are sent to the prediction coefficient 
memory 1 4, and are memorized. 

[0028] First, they are x1, xn about the transform coefficient for 1 block of the image in front 
of the zoom blocked as a criteria image. It carries out and they are these transform coefficients 

xl xn. They are q1 qn about the re-quantization data of the result of having performed p 

bit ADRC to each data. The class code class for every block when carrying out is defined by (1) 

type. For example, the above-mentioned transform coefficients x1 xn It is equivalent to 1 

block (n= 4) which consists of 4x4 pixels of (a) of drawin g 2 . 

[0029] 

[Equation 1] 

class ^j^qi&P)' 1 (1) 

[0030] Then, they are x1, xn about the transform coefficient of the block in the criteria image 
in front of the zoom which the transform coefficient which should generally be filled up within the 
expansion image expanded twice is set to y, and is supplemented with this transform coefficient 
y. Multipliers w1, wn shown in (2) types for every class classification when it carries out The 
linearity presumption type of n tap to depend is set up. 
y=w1 x1+w2 x2+ ... +wn xn ... (2) 

Before study, it is wi. It is an undetermined coefficient. 

[0031] Study is performed to two or more transform coefficients by which the class 
classification was carried out. For example, (2) types are followed when the number of transform 
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coefficients is m. yk =w1 xk1+w2 xk2+ ... +wn xkn ... (3) 
( — k= — it becomes 1, 2, m). 

[0032] Here, in m>n, they are multipliers w1 wn. Since it is not decided that it will be a 

meaning, it is [ ... (4) ] the element of the error vector e ek =yk-{w1 xk1+w2 xk2+ ... +wn xkn] 

( — k= — it is defined as 1 , 2 m), and asks for the multiplier which makes (5) types min. 

[0033] 
[Equation 2] 

m 2 

* 2 = ( 5 > 

k=l 

[0034] This is a solution method by the so-called least square method. Here, it is the (4)-type 
prediction coefficient set wi. It asks for the partial differential coefficient to depend. 
[0035] 
[Equation 3] 

aWi k=i 



m 



e k = ^2x kr e k 



(6) 



[0036] (6) It is each prediction coefficient set wi so that a formula may be set to 0. Since what is 
necessary is just to decide, it is [0037]. 
[Equation 4] 



m 



x ji = lL x ki' x 



(7) 



m 



(8) 



Yi = lL x ki'yk 



[0038] It is [0039], when it carries out and a matrix is used. 
[Equation 5] 



22 



2n 



X nl X 



n2 



nn 



J 



• • « 



• • • 



(9) 



[0040] It becomes. Generally this equation is called the normal equation, this equation — 
sweeping out — general matrix solution methods, such as law, — using — prediction coefficient 
set wi ******** — if it solves — prediction coefficient set wi It is decided. Therefore, it is the 
above-mentioned prediction coefficient set wi, using a class code as the address at the above- 
mentioned ROM table 5. It memorizes. 

[0041] Therefore, if the class code from the above-mentioned class code generating section 4 is 
inputted into the above-mentioned ROM table 5 when the transform coefficient for the 
supplement which it is beforehand learned in the above-mentioned ROM table 5, and is 
memorized is a prediction coefficient, the prediction coefficient data according to the inputted 
class code will be read. The prediction coefficient by which reading appearance was carried out 
[ above-mentioned ] is sent to the multiplier composition section 6. In this multiplier composition 



http://www4.ipdl jpo.gojp/cgi-bin/tran_web_cgLejje 



2004/08/27 



6/7 ^— v 



section 6, transform coefficient y' supplemented in an expansion image is computed by 
performing the operation which followed the prediction type of (10) types using the above- 
mentioned prediction coefficient. 
[0042] 

y-w1 x1+w2 x2+ ... +wn xn ... (10) 

[0043] Furthermore, the transform coefficient obtained in the above-mentioned orthogonal 
transformation section 3 is supplied to this multiplier composition section 6, the transform 
coefficient of the image data of this criteria image and the transform coefficient filled up are 
compounded, and the inverse transformation section 7 is supplied. In this inverse transformation 
section 7, the image data by which the expansion image was blocked is obtained by performing 
inverse transformation to the transform coefficient of the compounded expansion image. 
[0044] The image data of the expansion image obtained in the above-mentioned inverse 
transformation section 7 is sent to the block decomposition section 8, each block is 
decomposed, and the image data of one expansion image is generated. The image data of this 
expansion image is outputted from an output terminal 9. 

[0045] In addition, although ADRC was used as the image data compression approach for class 
division, the data compression approaches, such as VQ (vector quantization) and DPCM 
(predicting coding), may be used instead of this ADRC. 
[0046] 

[Effect of the Invention] The electronic zoom equipment concerning this invention so that clearly 
also from the above explanation A blocking means to block the image data of the criteria image 
inputted, An orthogonal transformation means to perform orthogonal transformation to the image 
data blocked from the above-mentioned blocking means, and to ask for a transform coefficient, 
A class classification means to generate the class code which classifies the transform 
coefficient from the above-mentioned orthogonal transformation means into two or more 
classes, and shows this class, A multiplier data generating means by which the transform 
coefficient data corresponding to the class code from the above-mentioned class classification 
means are read, A multiplier composition means to compound the transform coefficient of an 
expansion image with the transform coefficient from the above-mentioned orthogonal 
transformation means, and the transform coefficient data from the above-mentioned multiplier 
data generating means, Inverse transformation is performed to a transform coefficient from the 
above-mentioned multiplier composition means, and it consists of an inverse transformation 
means to output the image data for every block of an expansion image. For the above-mentioned 
multiplier data generating means It asks for a transform coefficient by blocking the image data of 
a criteria image and an expansion image, respectively, and performing orthogonal transformation. 
It asks for the transform coefficient data optimized by learning between the transform 
coefficient of the above-mentioned criteria image, and the transform coefficient of the above- 
mentioned expansion image. Since the transform coefficient data which the high frequency 
component of an expansion image reflected when the optimized transform coefficient data 
corresponding to the above-mentioned class code and this class code were beforehand 
memorized by the table are used, an expansion image can be compounded without reducing 
resolution. 

[0047] Moreover, a block chemically-modified [ which blocks the image data of the criteria image 
into which the electronic zoom approach concerning this invention is inputted ] degree, The 
orthogonal transformation process which performs orthogonal transformation to the image data 
blocked from the block chemically-modified [ above-mentioned ] degree, and asks for a 
transform coefficient, The class classification procedure which generates the class code which 
classifies the transform coefficient from the above-mentioned orthogonal transformation process 
into two or more classes, and shows this class. The multiplier data generating process that the 
transform coefficient data corresponding to the class code from the above-mentioned class 
classification procedure are read, The multiplier composition process which compounds the 
transform coefficient of an expansion image with the transform coefficient from the above- 
mentioned orthogonal transformation process, and the transform coefficient data from the 
above-mentioned multiplier data generating process, Inverse transformation is performed to a 
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transform coefficient from the above-mentioned multiplier composition process, and it consists 
of the inverse transformation process which outputs the image data for every block of an 
expansion image. At the above-mentioned multiplier data generating process It asks for a 
transform coefficient by blocking the image data of a criteria image and an expansion image, 
respectively, and performing orthogonal transformation. It asks for the transform coefficient data 
optimized by learning between the transform coefficient of the above-mentioned criteria image, 
and the transform coefficient of the above-mentioned expansion image. Since the transform 
coefficient data which the high frequency component of an expansion image reflected when the 
optimized transform coefficient data corresponding to the above-mentioned class code and this 
class code were beforehand memorized by the table are used, an expansion image can be 
compounded without reducing resolution. 



[Translation done.] 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely, 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is drawing showing the rough configuration of the electronic zoom equipment 
concerning this invention. 

[Drawing 2] It is drawing showing the configuration of the transform coefficient when carrying 
out orthogonal transformation of the configuration and expansion image of a transform 
coefficient when carrying out orthogonal transformation of the criteria image. 
[ Drawing 3] It is drawing showing a criteria image and an expansion image roughly. 
[Drawing 4] It is the flow chart which shows the study procedure of the transform coefficient 
data filled up in an expansion image. 

[ Drawi ng 5] It is drawing showing the rough configuration for study of a prediction coefficient. 
[Description of Notations] 

2 Blocking section 

3 Orthogonal transformation section 

4 Class code generating section 

5 ROM table 

6 Multiplier composition section 

7 Inverse transformation section 

8 Block decomposition section 

13 Study section 

14 Prediction coefficient memory 



[Translation done.] 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



CORRECTION OR AMENDMENT 



[Kind of official gazette] Printing of amendment by the convention of 2 of Article 1 7 of Patent 
Law 

[Section partition] The 3rd partition of the 7th section 
[Publication date] August 3, Heisei 13 (2001. 8.3) 

[Publication No.] JP,7-67031,A 

[Date of Publication] March 10, Heisei 7 (1995. 3.10) 
[Annual volume number] Open patent official report 7-671 
[Application number] Japanese Patent Application No. 5-213962 
[The 7th edition of International Patent Classification] 

H04N 5/262 

[FI] 

H04N 5/262 
[Procedure revision] 

[Filing Date] August 28, Heisei 12 (2000. 8.28) 

[Procedure amendment 1] 

[Document to be Amended] Specification 

[Item(s) to be Amended] Whole sentence 

[Method of Amendment] Modification 

[Proposed Amendment] 

[Document Name] Specification 

[Title of the Invention] An image processing system and the image-processing approach 
[Claim(s)] 

[Claim 1] A blocking means to block the image data of the criteria image inputted, 
An orthogonal transformation means to perform orthogonal transformation to the image data 
blocked from the above-mentioned blocking means, and to ask for a transform coefficient, 
A class classification means to generate the class code which classifies the transform 
coefficient from the above-mentioned orthogonal transformation means into two or more 
classes, and shows this class, 

A multiplier data generating means by which the transform coefficient data corresponding to the 
class code from the above-mentioned class classification means are read, 
A multiplier composition means to compound the transform coefficient of an expansion image 
with the transform coefficient from the above-mentioned orthogonal transformation means, and 
the transform coefficient data from the above-mentioned multiplier data generating means, 
The image processing system characterized by consisting of an inverse transformation means to 
perform inverse transformation to a transform coefficient from the above-mentioned multiplier 
composition means, and to output the image data for every block of an expansion image. 
[Claim 2] The image processing system according to claim 1 characterized by to ask for a 
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transform coefficient, to ask for the transform coefficient data optimized by learning between 
the transform coefficient of the above-mentioned criteria image, and the transform coefficient of 
the above-mentioned expansion image by blocking the image data of a criteria image and an 
expansion image for the above-mentioned multiplier data generating means, respectively, and 
performing orthogonal transformation to it, and for the optimized transform coefficient data 
corresponding to the above-mentioned class code and this class code to be beforehand 
memorized by the table. 

[Claim 3] A block chemically-modified [ which blocks the image data of the criteria image 
inputted ] degree, 

The orthogonal transformation process which performs orthogonal transformation to the image 
data blocked from the block chemically-modified [ above-mentioned ] degree, and asks for a 
transform coefficient, 

The class classification procedure which generates the class code which classifies the transform 
coefficient from the above-mentioned orthogonal transformation process into two or more 
classes, and shows this class, 

The multiplier data generating process that the transform coefficient data corresponding to the 

class code from the above-mentioned class classification procedure are read. 

The multiplier composition process which compounds the transform coefficient of an expansion 

image with the transform coefficient from the above-mentioned orthogonal transformation 

process, and the transform coefficient data from the above-mentioned multiplier data generating 

process, 

The image-processing approach characterized by consisting of the inverse transformation 
process which performs inverse transformation to a transform coefficient from the above- 
mentioned multiplier composition process, and outputs the image data for every block of an 
expansion image. 

[Claim 4] The image-processing approach according to claim 3 characterized by to ask for a 
transform coefficient, to ask for the transform coefficient data optimized by learning between 
the transform coefficient of the above-mentioned criteria image, and the transform coefficient of 
the above-mentioned expansion image by blocking the image data of a criteria image and an 
expansion image at the above-mentioned multiplier data generating process, respectively, and 
performing orthogonal transformation to it, and for the optimized transform coefficient data 
corresponding to the above-mentioned class code and this class code to be beforehand 
memorized by the table. 
[Detailed Description of the Invention] 
[0001] 

[Industrial Application] This invention relates to the image processing system and the image- 
processing approach of outputting the image with which the criteria image was expanded and the 
number of pixels was increased. 
[0002] 

[Description of the Prior Art] The number of pixels of the image which carried out a zoom of the 
digital image, was expanded, and was obtained with the television camera etc. will increase from 
the number of pixels of the original image. The method of expanding an image exists by extending 
the field by which orthogonal transformation is carried out to the approach of interpolating a 
pixel in an expansion image in space, and expanding an image to the approach, i.e., the approach 
to which the number of pixels is made increasing, of expanding such an image, filling up the 
transform coefficient when performing orthogonal transformation to the image data of an 
expansion image in conversion space in the field by which an escape was carried out [ above- 
mentioned ], and performing inverse transformation. 

[0003] Although the pixel is interpolated in the approach of interpolating the pixel in an 
expansion image, using a linear interpolation filter etc. in the above-mentioned space, resolution 
falls and that dotage is conspicuous poses a problem from the image before the expansion image 
obtained by this approach is expanded on the expansion image. 

[0004] The approach of filling up a transform coefficient and on the other hand performing 
inverse transformation does not reduce the resolution of an expansion image rather than the 
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approach of interpolating a pixel using the above-mentioned linear interpolation filter. 
[0005] 

[Problem(s) to be Solved by the Invention] By the way, the high frequency component of an 
image appears in the transform coefficient usually obtained by performing orthogonal 
transformation to image data. However, in the approach of filling up the conventional transform 
coefficient and performing inverse transformation, since the transform coefficient supplemented 
in the expansion image is only made into zero, the high frequency component is not contained. 
Therefore, although the number of pixels increases when these zero perform inverse 
transformation using the transform coefficient set up and filled up and compound an expansion 
image, the image for which resolution was insufficient is obtained. That is, improvement in the 
resolution for the number of pixels to have increased is not obtained with image expansion. 
[0006] Then, this invention can obtain an expansion image in view of the above-mentioned actual 
condition with the transform coefficient in which the description of an image was made to reflect 
including the high frequency component of an expansion image, and offers the image processing 
system and the image-processing approach that improvement in the resolution for the number of 
pixels to have increased is obtained. 
[0007] 

[Means for Solving the Problem] A blocking means to block the image data of the criteria image 
into which the image processing system concerning this invention is inputted, An orthogonal 
transformation means to perform orthogonal transformation to the image data blocked from the 
above-mentioned blocking means, and to ask for a transform coefficient, A class classification 
means to generate the class code which classifies the transform coefficient from the above- 
mentioned orthogonal transformation means into two or more classes, and shows this class, A 
multiplier data generating means by which the transform coefficient data corresponding to the 
class code from the above-mentioned class classification means are read, A multiplier 
composition means to compound the transform coefficient of an expansion image with the 
transform coefficient from the above-mentioned orthogonal transformation means, and the 
transform coefficient data from the above-mentioned multiplier data generating means, Inverse 
transformation is performed to a transform coefficient from the above-mentioned multiplier 
composition means, and the technical problem mentioned above by consisting of an inverse 
transformation means to output the image data for every block of an expansion image is solved. 
[0008] Moreover, it asks for a transform coefficient, it asks for the transform coefficient data 
optimized by learning between the transform coefficient of the above-mentioned criteria image, 
and the transform coefficient of the above-mentioned expansion image, and it is characterized 
by for the optimized transform coefficient data corresponding to the above-mentioned class 
code and this class code to be beforehand memorized by the table by blocking the image data of 
a criteria image and an expansion image for the above-mentioned multiplier data generating 
means, respectively, and performing orthogonal transformation to it. 

[0009] A block chemically-modified [ which blocks the image data of the criteria image into 
which the image-processing approach concerning this invention is inputted ] degree, The 
orthogonal transformation process which performs orthogonal transformation to the image data 
blocked from the block chemically-modified [ above-mentioned ] degree, and asks for a 
transform coefficient, The class classification procedure which generates the class code which 
classifies the transform coefficient from the above-mentioned orthogonal transformation process 
into two or more classes, and shows this class, The multiplier data generating process that the 
transform coefficient data corresponding to the class code from the above-mentioned class 
classification procedure are read, The multiplier composition process which compounds the 
transform coefficient of an expansion image with the transform coefficient from the above- 
mentioned orthogonal transformation process, and the transform coefficient data from the 
above-mentioned multiplier data generating process, Inverse transformation is performed to a 
transform coefficient from the above-mentioned multiplier composition process, and the 
technical problem mentioned above by consisting of the inverse transformation process which 
outputs the image data for every block of an expansion image is solved. 

[0010] Moreover, it asks for a transform coefficient, it asks for the transform coefficient data 
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optimized by learning between the transform coefficient of the above-mentioned criteria image, 
and the transform coefficient of the above-mentioned expansion image, and it is characterized 
by for the optimized transform coefficient data corresponding to the above-mentioned class 
code and this class code to be beforehand memorized by the table by blocking the image data of 
a criteria image and an expansion image at the above-mentioned multiplier data generating 
process, respectively, and performing orthogonal transformation to it. 
[0011] 

[Function] In this invention, an expansion image is compounded by compounding the transform 
coefficient data in the extended field which is beforehand learned by the transform coefficient 
and the multiplier data generating means which carried out orthogonal transformation of the 
image data of a criteria image, and it asked for it, and is memorized, and performing inverse 
transformation. 
[0012] 

[Example] Hereafter, the desirable example of this invention is explained, referring to a drawing. 
The rough configuration of the electronic zoom equipment used as the example of the image 
processing system concerning this invention is shown in drawing 1. 

[0013] The digital image data of the criteria image inputted from the input terminal 1 of drawing 1 
are sent to the blocking section 2. The rate of a zoom of the request which expands a criteria 
image is given from the outside to the above-mentioned blocking section 2. Therefore, in the 
above-mentioned blocking section 2, with the above-mentioned rate of a zoom, the field where it 
is expanded in a criteria image can be pinpointed, and blocking of the image data of this 
pinpointed field that should be expanded is performed. The image data by which blocking was 
carried out [ above-mentioned ] is outputted to the orthogonal transformation section 3 for 
every block. In this orthogonal transformation section 3, orthogonal transformation is performed 
to each image data for every above-mentioned block, and a transform coefficient is obtained. As 
this orthogonal transformation, it is possible to use DCT (Discrete Cosine Transform), a 
Hadamard transform, KL (karroo NENREBU) conversion, etc. 

[0014] For example, in the above-mentioned blocking section 2, when orthogonal transformation 
is performed to each pixel data blocked in four-line four trains (4x4), a transform coefficient 
group as shown by O mark of (a) of drawing 2 is obtained. 

[0015] Here, the image data blocked in four-line four trains in a criteria image (4x4) is doubled 
two, When compounding the expansion image expanded to the image data by which eight-line 
eight trains (8x8) were blocked, since the number of pixels increases to 8x8 from 4x4, it will be 
needed to make it increase similarly and a transform coefficient will also fill up the transform 
coefficient shown by x mark of (a) of drawing 2. Therefore, the location shown by - mark of (b) 
of drawing 2 is supplemented with a transform coefficient, and the image data by which the 
expansion image which expanded the criteria image twice was blocked can be obtained by 
performing inverse transformation to all the transform coefficients shown by O mark and - mark. 
That is, the image data from which the number of pixels was increased can be obtained. 
[0016] The transform coefficient obtained in the above-mentioned orthogonal transformation 
section 3 is sent to the class code generating section 4. In this class code generating section 4, 
the class classification of the transform coefficient for every sent block is carried out, and the 
class code of this transform coefficient by which the class classification was carried out is 
generated. The generated class code is sent to the ROM (read-only memory) table 5. 
[0017] The optimized transform coefficient data with which this ROM table 5 is supplemented in 
an expansion image are beforehand obtained by study, and are memorized as a table with the 
class code. Therefore, the transform coefficient data in the expansion image corresponding to 
this class code are read from the above-mentioned ROM table 5 by using the class code from 
the above-mentioned class code generating section 4 as the address. 

[001 8] Here, the transform coefficient data beforehand learned by the above-mentioned ROM 
table 5 are explained. 

[0bl9] First, for example, the criteria image of (a) of drawing 3 and the expansion image which 
shows the zoom core s of this criteria image to (b) of drawing 3 expanded as a core are 
prepared. It can ask for the image data of this expansion image by the rate calculation approach 
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of a zoom proposed in a specification, a drawing, etc. of JP.4-318766.A. 

[0020] The procedure when learning a transform coefficient is shown in the flow chart of drawing 
4, and at step SI, the image data of the criteria image of (a) of drawing 3 is blocked, orthogonal 
transformation is performed to the image data by which blocking was carried out [ above- 
mentioned ] at step S2, and it asks for a transform coefficient. Furthermore, at step S3, the 
class classification of the above-mentioned transform coefficient is carried out, and the class 
code corresponding to this transform coefficient by which the class classification was carried 
out is generated. 

[0021] Moreover, it is parallel to above-mentioned actuation, and in step S4, the image data of 
the expansion image of (b) of drawing 3 is blocked, orthogonal transformation is performed to the 
image data by which blocking was carried out [ above-mentioned ] at step S5, and it asks for a 
transform coefficient. 

[0022] Next, it progresses to step S6 and learns between the transform coefficient by which the 
class classification was carried out at step S3, and the transform coefficient of the expansion 
image corresponding to this transform coefficient. Then, the class classification of the transform 
coefficient is called for and carried out about all the blocked image data in a criteria image at 
step S7, and it distinguishes whether study was performed between the transform coefficients of 
the image data by which the expansion image was blocked. 

[0023] It learns by blocking the image data of the criteria image with which it returns to step S1 
and step S4, and study is not performed further yet if it becomes and expansion image with 
which the above-mentioned study is performed to no blocked image data, performing orthogonal 
transformation to this blocked image data, and asking for a transform coefficient. However, if the 
above-mentioned study is performed to all the blocked image data, a mapping table will be 
generated from the transform coefficient by which the expansion image corresponding to the 
class code corresponding to the class classification by which study was carried out [ above- 
mentioned ] by progressing to step S8, and this class code was optimized. Thus, into the above- 
mentioned ROM table 5, a predetermined class code and the transform coefficient data 
corresponding to this class code are learned beforehand, and two or more storage is carried out. 
[0024] The simplest classification approach of the class code memorized in the above- 
mentioned ROM table 5 is an approach which does not process compression etc. to the 
transform coefficient obtained by performing orthogonal transformation to the image data for 
every block of the criteria image by which a class classification is carried out, but uses the 
transform coefficient of the image data for every above-mentioned block as a class code as it is. 
However, when a transform coefficient is a class code as it is, the pattern of a number 
equivalent to the above-mentioned class code of transform coefficients must be memorized, 
and, for that, ROM of a huge capacity is needed. Therefore, when implementability is taken into 
consideration, it is possible to use compressive processing for the transform coefficient of the 
criteria image learned. 

[0025] How to reduce for example, the above-mentioned number of class codes there using 
ADRC (ecad dynamic range coding) can be considered. Originally, ADRC js the accommodative 
re-quantizing method developed for [ to turn ] high efficiency coding VTR (video tape recorder), 
and can express the local pattern of signal level efficiently by the short word length. Therefore, 
when ADRC is used, the number of the effective class codes which saved the property of the 
pattern of a transform coefficient can be reduced. 

[0026] Here, the study approach of a prediction coefficient of having used ADRC for the class 
division in the case of using a prediction coefficient as the above-mentioned transform 
coefficient is explained concretely. 

[0027] Drawing 5 shows the rough configuration at the time of learning the prediction coefficient 
memorized in the above-mentioned ROM table 5. From the input terminal 11 of this drawing 5, 
the transform coefficient obtained by performing orthogonal transformation to the 1 -block image 
data of the criteria image which is an image in front of the blocked zoom is inputted, and the 
transform coefficient obtained by performing orthogonal transformation to the 1 -block image 
data of the blocked expansion image corresponding to the 1 -block image data of the above- 
mentioned criteria image is inputted from an input terminal 12. The transform coefficient of 
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these two blocks is sent to the study section 13. In this study section 13, it learns using the 
transform coefficient of the inputted criteria image, and the transform coefficient of an 
expansion image, and the prediction coefficient corresponding to a class code and this class 
code is created. This class code and prediction coefficient are sent to the prediction coefficient 
memory 14, and are memorized. 

[0028] First, they are x1 xn about the transform coefficient for 1 block of the image in front 

of the zoom blocked as a criteria image. It carries out and they are these transform coefficients 

x1, .... xn. They are q1 qn about the re-quantization data of the result of having performed p 

bit ADRC to each data. The class code class for every block when carrying out is defined by (1) 
type. For example, the above-mentioned transform coefficients x1, xn It is equivalent to 1 
block (n= 4) which consists of 4x4 pixels of (a) of drawing 2. 
[0029] 
[Equation 1] 



[0030] Then, they are x1, xn about the transform coefficient of the block in the criteria image 
in front of the zoom which the transform coefficient which should generally be filled up within the 
expansion image expanded twice is set to y, and is supplemented with this transform coefficient 

y. Multipliers w1 wn shown in (2) types for every class classification when it carries out The 

linearity presumption type of n tap to depend is set up. 
[0031] 

y=w1 x1+w2 x2+ ... +wn xn ... (2) 

Before study, it is wi. It is an undetermined coefficient. 

[0032] Study is performed to two or more transform coefficients by which the class 
classification was carried out. For example, (2) types are followed when the number of transform 
coefficients is m, 

yk =w1 xk1+w2 xk2+ ... +wn xkn ... (3) 

(k=1,2 m) 

It becomes. 

[0033] Here, in m>n, they are multipliers w1, wn. Since it is not decided that it will be a 
meaning, it is the element of the error vector e, 
ek =yk-{w1 xk1+w2 xk2+ ... +wn xkn} ... (4) 
(k=1,2, m) 

A definition is given and it asks for the multiplier which makes (5) types min. 

[0034] 

[Equation 2] 




i-1 




m 



2 





[0035] This is a solution method by the so-called least square method. Here, it is the (4)-type 
prediction coefficient set wi. It asks for the partial differential coefficient to depend. 
[0036] 
[Equation 3] 
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^ 2 = y 2 
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fe=i 



(6) 



[0037] (6) It is each prediction coefficient set wi so that a formula may be set to 0. Since what is 
necessary is just to decide, 
[0038] 
[Equation 4] 
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[0039] It is, when it carries out and a matrix is used, 

[0040] 

[Equation 5] 
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[0041] It becomes. Generally this equation is called the normal equation, this equation — 
sweeping out — general matrix solution methods, such as law, — using — prediction coefficient 
set wi ******** — if it solves — prediction coefficient set wi It is decided. Therefore, it is the 
above-mentioned prediction coefficient set wi, using a class code as the address at the above- 
mentioned ROM table 5. It memorizes. 

[0042] Therefore, if the class code from the above-mentioned class code generating section 4 is 
inputted into the above-mentioned ROM table 5 when the transform coefficient for the 
supplement which it is beforehand learned in the above-mentioned ROM table 5, and is 
memorized is a prediction coefficient, the prediction coefficient data according to the inputted 
class code will be read. The prediction coefficient by which reading appearance was carried out 
[ above-mentioned ] is sent to the multiplier composition section 6. In this multiplier composition 
section 6, transform coefficient y' supplemented in an expansion image is computed by 
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performing the operation which followed the prediction type of (10) types using the above- 
mentioned prediction coefficient. 
[0043] 

y-w1 x1+w2 x2+ ... +wn xn ... (10) 

Furthermore, the transform coefficient obtained in the above-mentioned orthogonal 
transformation section 3 is supplied to this multiplier composition section 6, the transform 
coefficient of the image data of this criteria image and the transform coefficient filled up are 
compounded, and the inverse transformation section 7 is supplied. In this inverse transformation 
section 7, the image data by which the expansion image was blocked is obtained by performing 
inverse transformation to the transform coefficient of the compounded expansion image. 
[0044] The image data of the expansion image obtained in the above-mentioned inverse 
transformation section 7 is sent to the block decomposition section 8, each block is 
decomposed, and the image data of one expansion image is generated. The image data of this 
expansion image is outputted from an output terminal 9. 

[0045] In addition, although ADRC was used as the image data compression approach for class 
division, the data compression approaches, such as VQ (vector quantization) and DPCM 
(predicting coding), may be used instead of this ADRC. 
[0046] 

[Effect of the Invention] The image processing system applied to this invention so that clearly 
also from the above explanation, A blocking means to block the image data of the criteria image 
inputted, An orthogonal transformation means to perform orthogonal transformation to the image 
data blocked from the above-mentioned blocking means, and to ask for a transform coefficient, 
A class classification means to generate the class code which classifies the transform 
coefficient from the above-mentioned orthogonal transformation means into two or more 
classes, and shows this class, A multiplier data generating means by which the transform 
coefficient data corresponding to the class code from the above-mentioned class classification 
means are read, A multiplier composition means to compound the transform coefficient of an 
expansion image with the transform coefficient from the above-mentioned orthogonal 
transformation means, and the transform coefficient data from the above-mentioned multiplier 
data generating means, Inverse transformation is performed to a transform coefficient from the 
above-mentioned multiplier composition means, and it consists of an inverse transformation 
means to output the image data for every block of an expansion image. For the above-mentioned 
multiplier data generating means It asks for a transform coefficient by blocking the image data of 
a criteria image and an expansion image, respectively, and performing orthogonal transformation. 
It asks for the transform coefficient data optimized by learning between the transform 
coefficient of the above-mentioned criteria image, and the transform coefficient of the above- 
mentioned expansion image. Since the transform coefficient data which the high frequency 
component of an expansion image reflected when the optimized transform coefficient data 
corresponding to the above-mentioned class code and this class code were beforehand 
memorized by the table are used, an expansion image can be compounded without reducing 
resolution. 

[0047] Moreover, the image-processing approach concerning this invention is a block 
chemically-modified [ which blocks the image data of the criteria image inputted ] degree, The 
orthogonal transformation process which performs orthogonal transformation to the image data 
blocked from the block chemically-modified [ above-mentioned ] degree, and asks for a 
transform coefficient, The class classification procedure which generates the class code which 
classifies the transform coefficient from the above-mentioned orthogonal transformation process 
into two or more classes, and shows this class, The multiplier data generating process that the 
transform coefficient data corresponding to the class code from the above-mentioned class 
classification procedure are read, The multiplier composition process which compounds the 
transform coefficient of an expansion image with the transform coefficient from the above- 
mentioned orthogonal transformation process, and the transform coefficient data from the 
above-mentioned multiplier data generating process, Inverse transformation is performed to a 
transform coefficient from the above-mentioned multiplier composition process, and it consists 

>. 
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of the inverse transformation process which outputs the image data for every block of an 
expansion image. At the above-mentioned multiplier data generating process It asks for a 
transform coefficient by blocking the image data of a criteria image and an expansion image, 
respectively, and performing orthogonal transformation. It asks for the transform coefficient data 
optimized by learning between the transform coefficient of the above-mentioned criteria image, 
and the transform coefficient of the above-mentioned expansion image. Since the transform 
coefficient data which the high frequency component of an expansion image reflected when the 
optimized transform coefficient data corresponding to the above-mentioned class code and this 
class code were beforehand memorized by the table are used, an expansion image can be 
compounded without reducing resolution. 
[Brief Description of the Drawings] 

[Drawing 1] It is drawing showing the rough configuration of the electronic zoom equipment used 

as the example of the image processing system concerning this invention. 

[Drawing 2] It is drawing showing the configuration of the transform coefficient when carrying 

out orthogonal transformation of the configuration and expansion image of a transform 

coefficient when carrying out orthogonal transformation of the criteria image. 

[Drawing 3] It is drawing showing a criteria image and an expansion image roughly. 

[Drawing 4] It is the flow chart which shows the study procedure of the transform coefficient 

data filled up in an expansion image. 

[Drawing 5] It is drawing showing the rough configuration for study of a prediction coefficient. 
[Description of Notations] 

2 Blocking Section 3 Orthogonal Transformation Section 4 Class Code Generating Section 5 
ROM Table 6 Multiplier Composition Section 7 Inverse Transformation Section 8 Block 
Decomposition Section, 13 Study Section 14 Prediction Coefficient Memory 



[Translation done.] 
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